Two-Layer Contractive Encodings with Linear Transformation of Perceptrons for Semi-Supervised Learning
نویسندگان
چکیده
It is difficult to train a multi-layer perceptron (MLP) when there are only a few labeled samples available. However, by pretraining an MLP with vast amount of unlabeled samples available, we may achieve better generalization performance. Schulz et al. (2012) showed that it is possible to pretrain an MLP in a less greedy way by utilizing the two-layer contractive encodings, however, with a cost of a more difficult optimization problem. On the other hand, Raiko et al. (2012) proposed a scheme for making the optimization problem much easier in deep networks. In this paper, we show that it is beneficial to combine these two approaches.
منابع مشابه
Two-Layer Contractive Encodings with Shortcuts for Semi-supervised Learning
Supervised training of multi-layer perceptrons (MLP) with only few labeled examples is prone to overfitting. Pretraining an MLP with unlabeled samples of the input distribution may achieve better generalization. Usually, pretraining is done in a layer-wise, greedy fashion which limits the complexity of the learnable features. To overcome this limitation, two-layer contractive encodings have bee...
متن کاملTwo-layer contractive encodings for learning stable nonlinear features
Unsupervised learning of feature hierarchies is often a good strategy to initialize deep architectures for supervised learning. Most existing deep learning methods build these feature hierarchies layer by layer in a greedy fashion using either auto-encoders or restricted Boltzmann machines. Both yield encoders which compute linear projections of input followed by a smooth thresholding function....
متن کاملLearning Two-Layer Contractive Encodings
Unsupervised learning of feature hierarchies is often a good initialization for supervised training of deep architectures. In existing deep learning methods, these feature hierarchies are built layer by layer in a greedy fashion using auto-encoders or restricted Boltzmann machines. Both yield encoders, which compute linear projections followed by a smooth thresholding function. In this work, we...
متن کاملA Comparison of Discriminative EM-Based Semi-Supervised Learning algorithms on Agreement/Disagreement classification
Recently, semi-supervised learning has been an active research topic in the natural language processing community, to save effort in hand-labeling for data-driven learning and to exploit a large amount of readily available unlabeled text. In this paper, we apply EM-based semi-supervised learning algorithms such as traditional EM, co-EM, and cross validation EM to the task of agreement/disagreem...
متن کاملA Semi-Supervised Learning Algorithm for Multi-Layered Perceptrons
We address the issue of learning multi-layered perceptrons (MLPs) in a discriminative, inductive, multiclass, parametric, and semi-supervised fashion. We introduce a novel objective function that, when optimized, simultaneously encourages 1) accuracy on the labeled points, 2) respect for an underlying graph-represented manifold on all points, 3) smoothness via an entropic regularizer of the cla...
متن کامل